Benchmarking and Configuration of Workflow Management Systems

Workflow management systems are a cornerstone of mission-criticial, possibly cross-organizational business processes. For large-scale applications both their performance and availability are crucial factors, and the system needs to be properly configured to meet the application demands. Despite ample work on scalable system architectures for workflow management, the literature has neglected the important issues of how to systematically measure the performance of a given system configuration and how to determine viable configurations without resorting to expensive trial-and-error or guesswork. This paper proposes a synthetic benchmark for workflow management systems; based on the TPCC order-entry benchmark, a complete e-commerce workflow is specified in a system-independent form. This workflow benchmark, which stresses all major components of a workflow system and is parameterized in a flexible manner, has been applied to two operational systems, the commercial system Staffware97 and our own prototype system Mentor-lite. The paper reports performance results from our measurements and discusses lessons learned. In particular, the results demonstrate the scalability of the Mentor-lite architecture. The measurements also underline the need for configuring systems intelligently, and the paper outlines an auto-configuration tool that we have been building to this end.